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OPTICAL ENCODING OF AUDIO DATA 



FIELD OF THE INVENTION 

[0001] The present invention relates to encoding data and more particularly to 
manipulating audio data so that it can be encoded along with video data. 

DESCRIPTION OF THE RELATED ART 

[0002] Typically a movie includes a sequence of video frames together with a 
corresponding sequence of audio frames (i.e., a video track and an audio track). 
Synchronization of these frames on playback is crucial for an audience's appreciation of the 
movie. However, these sequences are generally processed separately because of characteristic 
differences between video and audio data. Compression is an example of a processing step that 
is performed separately for video and audio data. 

[0003] The nature of video data requires that compression be performed separately. 
Video data is typically a frame corresponding to a two-dimensional display. For example, a 
DVD (Digital Video Disk) typically employs a 720x480 array of pixels where each pixel 
contains a multi-bit value, such as 16-bit, 24-bit or 32-bit, that corresponds to an enumerated 
color. 

[0004] Audio data on the other hand, is typically time-varying waveform data that 
represents a voltage or current rather than color. The data can be 16-bit values or higher bit 
values that correspond to the voltage or current that will drive a speaker. 

[0005] Because of these characteristic differences, separate encoders and decoders are 
used for video and audio data. Having two separate encoders and decoders is an inefficient use 
of resources and costly. Further, synchronization between the separate encoders and decoders 
may not be maintained. It would, therefore, be desirable to use one encoder and decoder for 
both video and audio data. The present invention provides a mechanism for allowing audio data 
to be manipulated so that it can be concurrently encoded and decoded with video data. 
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SUMMARY OF THE INVENTION 
[0006] A method for representing audio data in a format that can be operated 

upon independently, or merged with video data. The method includes replacing each audio 
information element in an audio sequence with a corresponding color from a color palette. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0007] The present invention is illustrated by way of example, and not limitation, in the 
figures of the accompanying drawings in which like references denote similar elements, and in 
which: 

Figure la illustrates a representative audio signal; 

Figure lb illustrates a representative digitally sampled audio signal; 

Figure 2 illustrates graphically a digitally sampled audio signal being mapped to colors 
selected from a palette of possible colors; 

Figure 3 illustrates a process for mapping a digitally sampled audio signal to colors 
selected from a palette of possible colors; and 

Figure 4 illustrates a process for recovering the audio frame from the color audio frame. 
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DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS OF THE INVENTION 



[0008] Methods and apparatus for manipulating audio data so that it may be encoded and 
decoded along with video data are described. In the following description, for purposes of 
explanation, numerous specific details are set forth in order to provide a thorough understanding 
of the present invention. It will be evident, however, to one skilled in the art that the present 
invention may be practiced with a variety of data, especially audio and video, without these 
specific details. In other instances, well-known operations, steps, functions and elements are not 
shown in order to avoid obscuring the invention. 

[0009] Various operations will be described as multiple discrete steps performed in turn 
in a manner that is most helpful in understanding the present invention. However, the order of 
description should not be construed as to imply that these operations are necessarily performed 
in the order that they are presented, or even order dependent. Lastly, repeated usage of the 
phrases "in one embodiment," "an alternative embodiment," or an "alternate embodiment" does 
not necessarily refer to the same embodiment, although it may. 

[0010] Figure la illustrates a representative audio signal Before an audio signal can be 
digitally encoded and transmitted it needs to be transformed into a digital signal, although 
implementation of the present invention will typically occur on audio signals that have 
previously been transformed into digital signals. To transform audio signal 100 into a digital 
signal, audio signal 100 is typically sampled by an analog to digital converter at a predetermined 
rate to produce snapshots of the value of the audio signal at equally spaced intervals, as is 
conventionally known. Depending on the audio scheme being implemented a certain number of 
samples make up a frame. Typically, samples are encoded or processed using frames. 

[0011] Figure lb illustrates a representative digitally sampled audio signal. Digitally 
sampled audio signal 104 is a sequence of digital values, also termed digital audio signal 
elements, that are spaced apart by the same time interval. The sequence of digital audio signal 
elements can be represented in a two column table in which each row contains the time a sample 
was taken and the digital value of the sampled audio signal at the sample time. Table 106 shows 
such a table or data. 

[0012] Since audio and video data have different formats, audio data is not 
conventionally appended to video data and encoded with it. The present invention provides a 
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mechanism for manipulating audio data so that it can be appended to video data for later 
encoding concurrently with the video data. 

[0013] Figure 2 illustrates graphically a digitally sampled audio signal being mapped to 
colors selected from a palette of possible colors. Audio data from various points in time, each 
audio signal element in other words, is tracked in time based upon a header (not shown) that 
indicates the playback rate, which then allows playback of the sequence of digital audio signal 
elements at the appropriate time. All of the digital audio signal elements that occur at different 
points in time that have the same amplitude have the same color assigned to them. The process 
of mapping assigns a color to the corresponding digital audio signal element at each different 
point in time, as shown at 204. After the process of mapping, each of the digital audio signal 
elements, instead of having an associated amplitude, has an associated color obtained from a 
color lookup table. Audio signals that have the same amplitudes will thus have the same color. 
For example, ti, t 7 , and t 2 2 all have the same color assigned to them from the palette 200. 
Similarly, t 2 and t2o have the same pointer, 1, assigned to them. The color assigned to a 
particular amplitude is thus a function of the amplitude. Palette 200 is a sub-palette of the 
palette of possible colors. 

[0014J Figure 3 illustrates a process for mapping a digital audio signal element to a color 
selected from a palette of possible colors. According to process 300, the amplitude for a digital 
audio signal element is read in at 302. At 304, it is determined whether a color has been 
previously assigned to the amplitude. If a color has been previously assigned to the amplitude, 
the previously assigned color lookup for the color is assigned to the current element sample at 
306. If a color has not been previously assigned to the amplitude, a new color lookup is 
assigned to both the color and amplitude and the color is added to a sub-palette at 308. The sub- 
palette is the set of colors that have been assigned to the amplitudes of the digitally sampled 
audio signal elements. At 310, it is determined whether there are any more digital audio signal 
elements to process. If there are more digital audio signal elements to process, process 300 
advances to the next sample at 3 12 and the amplitude for the current sample is read in at 302. At 
the end of process 300, the sub-palette contains all the colors that were needed to describe the 
amplitudes at all the times of the digitally sampled audio signal elements. Also for each sample 
in the frame, instead of an amplitude there is an associated color from the sub-palette. The 
output of process 300 are a frame that contains the sub-palette and the sequence digital audio 
signals in their transformed color format. 
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[0015] The color audio frame of process 300 is added to a corresponding video frame to 
produce an augmented video frame that can be encoded and later decoded. In order to not 
obscure the present invention, methods and apparatus of adding the color audio frame to a 
corresponding video frame, and then operating upon the augmented frame will not be described 
in greater detail Conventional methods, as well as methods described in co-pending 
applications entitled "Method And Apparatus For Determining Patterns Within Adjacent Blocks 
Of Data bearing attorney reference number 042503/0259665 and filed on the same day as this 
application in the U.S. Patent Office, and U.S. Appln. No. 09/727,096 entitled "Method And 
Apparatus For Encoding Information Using Multiple Passes And Decoding In A Single Pass" 
filed on November 29, 2000, both of which are assigned to the same assignee as the present 
invention, can be used for adding the color audio frame to the video frame to obtain the 
augmented frame and then operating upon the augmented frame, although it will be understood 
that the present invention can be used independently and without reference to a video frame. 

[0016] After the augmented video frame is decoded, the color audio can be separated 
from the augmented video frame and the digital audio signal elements recovered. Figure 4 
illustrates a process for recovering the digital audio signal elements. According to process 400, 
the digital color value for the current digital audio signal element is read in at 402 and the 
corresponding digital audio value is retrieved based upon the color lookup at 404. At 406, it is 
determined whether there are any more digital color values representing the digital audio signal 
elements to process. If there are more values to process, at 408 the process advances to the next 
sample and the amplitude for the now current sample is read in at 402. The output of process 
400 are the original digitally sampled audio signals. 

[0017] Thus, methods and apparatus for manipulating audio data so that it may be 
encoded independently in a different form, and, more preferably, along with video data have 
been described. Although the present invention has been described with reference to specific 
exemplary embodiments, it will be evident to one of ordinary skill in the art that various 
modifications and changes may be made to these embodiments without departing from the 
broader spirit and scope of the invention as set forth in the claims. Accordingly, the 
specification and drawings are to be regarded in an illustrative rather than a restrictive sense. 
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